The following project is a taste of the power StyleGAN 2 has to offer, in terms of Image Reconstruction problems.
In this project I dealth with reconstructing images with different forms of degradations, e.g., Blurry / BW / masked images.
The following lines contains some outputs examples after running the model using different modes, depending on the original image.
Hope you'll find it useful and have some fun using it!
For this section, I implemented a basic function which returns a greyscale image, given RGB images. The process was as follows:
a) Take the input image and align it.
b) If the input image is not already corrupted, we'll call colorization_mode inside invert_image, and send the greyscale image as the "original" input image to run_latent_optimization.
c) Inside the last function, after each iteration the NN outputs a synthetic output image. To make the NN deliver a colorized reconstruction of the image, we will take the synthetized image the GAN is generated on each step and convert it to a greyscale image using the colorization_mode before calculating the loss function.
## Alan Turing's Colorization
tar = "/content/gdrive/MyDrive/input_img/inputs/alan_turing_grayscale.png"
outdir = "/content/gdrive/MyDrive/input_img/Alan"
alan_loss = invert_image(GRAYSCALE_DEGRADATION, tar,outdir, 'Alan Turing',num_steps=500,latent_dist_reg_weight=0.5 ,is_input_degraded=True)
## Alan Turing's Colorization
tar = "/content/gdrive/MyDrive/input_img/inputs/alan_turing_grayscale.png"
outdir = "/content/gdrive/MyDrive/input_img/Alan/with\ 500\ and\ 0.01"
alan_loss = invert_image(GRAYSCALE_DEGRADATION, tar,outdir, 'Alan Turing',num_steps=500,latent_dist_reg_weight=0.01 ,is_input_degraded=True)
## Alan Turing's Colorization
tar = "/content/gdrive/MyDrive/input_img/inputs/alan_turing_grayscale.png"
outdir = "/content/gdrive/MyDrive/input_img/Alan/with\ 600\ and\ 0.001"
alan_loss = invert_image(GRAYSCALE_DEGRADATION, tar,outdir, 'Alan Turing',num_steps=600,latent_dist_reg_weight=0.001 ,is_input_degraded=True)
## Alan Turing's Colorization
tar = "/content/gdrive/MyDrive/input_img/inputs/alan_turing_grayscale.png"
outdir = "/content/gdrive/MyDrive/input_img/Alan/with\ 1500\ and\ 0.2"
alan_loss = invert_image(GRAYSCALE_DEGRADATION, tar,outdir, 'Alan Turing',num_steps=700,latent_dist_reg_weight=0.2 ,is_input_degraded=True)
## Alan Turing's Colorization
tar = "/content/gdrive/MyDrive/input_img/inputs/alan_turing_grayscale.png"
outdir = "/content/gdrive/MyDrive/input_img/Alan/with\ 1500\ and\ 0.01"
alan_loss = invert_image(GRAYSCALE_DEGRADATION, tar,outdir, 'Alan Turing',num_steps=1500,latent_dist_reg_weight=0.9 ,is_input_degraded=True)
## Tzlil's Colorization:
tar = "/content/gdrive/MyDrive/input_img/Tzlil/africa_tzlil_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tzlil/colorization/with 900 and 0.1"
tzlil_loss = invert_image(GRAYSCALE_DEGRADATION, tar,outdir, "Tzlil Colorization", num_steps=900, latent_dist_reg_weight=0.1)
## Tzlil's Colorization:
tar = "/content/gdrive/MyDrive/input_img/Tzlil/africa_tzlil_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tzlil/colorization/with 900 and 0.5"
tzlil_loss = invert_image(GRAYSCALE_DEGRADATION, tar,outdir, "Tzlil Colorization", num_steps=1200, latent_dist_reg_weight=0.5)
## Tzlil's Colorization:
tar = "/content/gdrive/MyDrive/input_img/Tzlil/africa_tzlil_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tzlil/colorization/with 900 and 0.3"
tzlil_loss = invert_image(GRAYSCALE_DEGRADATION, tar,outdir, "Tzlil Colorization", num_steps=900, latent_dist_reg_weight=0.3)
## Tzlil's Colorization:
tar = "/content/gdrive/MyDrive/input_img/Tzlil/africa_tzlil_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tzlil/colorization/with 1000 and 1"
tzlil_loss = invert_image(GRAYSCALE_DEGRADATION, tar,outdir, "Tzlil Colorization", num_steps=1000, latent_dist_reg_weight=1)
## Tzlil's Colorization:
tar = "/content/gdrive/MyDrive/input_img/Tzlil/africa_tzlil_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tzlil/colorization/with 900 and 0.001"
tzlil_loss = invert_image(GRAYSCALE_DEGRADATION, tar,outdir, "Tzlil Colorization", num_steps=900, latent_dist_reg_weight=0.001)
## Tirtza's Colorization:
tar = "/content/gdrive/MyDrive/input_img/Tirtsa/tir_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tirtsa/colorization/with 900 and 0.4 "
tirtz_loss = invert_image(GRAYSCALE_DEGRADATION, tar,outdir, "Tirtsa Colorization",num_steps=900,latent_dist_reg_weight=0.4)
## Tirtza's Colorization:
tar = "/content/gdrive/MyDrive/input_img/Tirtsa/tir_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tirtsa/colorization/with 900 and 0.6 "
tirtz_loss = invert_image(GRAYSCALE_DEGRADATION, tar,outdir, "Tirtsa Colorization",num_steps=900,latent_dist_reg_weight=0.6)
For this section, I implemented a blurring function for a RGB images. The process was as follows:
a) Take the input image and align it.
b) Call blur_mode inside invert_image, and send the blurred image as the "original" input image to run_latent_optimization.
c) Inside the last function, after each iteration the NN outputs a synthetic output image. In order to make the NN result in a deblurred image, we will take the synthetized image, and blur it using the blur_mode before calculating the loss function.
## Yann Lecun's Deblurring
tar = "/content/gdrive/MyDrive/input_img/inputs/yann_lecun_blur.png"
outdir = "/content/gdrive/MyDrive/input_img/yann/with 700 and 0.5"
yann_loss = invert_image(GAUSSIAN_BLUR_DEGRADATION, tar,outdir,'Yann Lecun' ,num_steps=700,latent_dist_reg_weight=0.5, is_input_degraded=True)
## Yann Lecun's Deblurring
tar = "/content/gdrive/MyDrive/input_img/inputs/yann_lecun_blur.png"
outdir = "/content/gdrive/MyDrive/input_img/yann/with 700 and 0.1"
yann_loss = invert_image(GAUSSIAN_BLUR_DEGRADATION, tar,outdir,'Yann Lecun' ,num_steps=700,latent_dist_reg_weight=0.1, is_input_degraded=True)
## Yann Lecun's Deblurring
tar = "/content/gdrive/MyDrive/input_img/inputs/yann_lecun_blur.png"
outdir = "/content/gdrive/MyDrive/input_img/yann/with 1000 and 0.001"
yann_loss = invert_image(GAUSSIAN_BLUR_DEGRADATION, tar,outdir,'Yann Lecun' ,num_steps=1000,latent_dist_reg_weight=0.001, is_input_degraded=True)
## Tzlil's reconstruction:
tar = "/content/gdrive/MyDrive/input_img/Tzlil/africa_tzlil_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tzlil/deblur"
tzlil_loss = invert_image(GAUSSIAN_BLUR_DEGRADATION ,tar,outdir, "Tzlil Deblurring", num_steps=1000,latent_dist_reg_weight=0.2)
# Using ldrw = 0.1 got creepy results
## Tirtza's reconstruction:
tar = "/content/gdrive/MyDrive/input_img/Tirtsa/tir_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tirtsa/deblur"
tirtz_loss = invert_image(GAUSSIAN_BLUR_DEGRADATION, tar,outdir, "Tirtsa Deblurring", num_steps=700,latent_dist_reg_weight=0.2)
For this section, I implemented a function used to mask the image in a manner which mimics the making we asked to perform. The way I implemented it was as follows:
• Estimate what are the boundaries of the original masking provided for this exercise.
• Set all the pixels which falls inside the boundaries to zero. From here, all we got left to do is to figure out is where we should call the degrading function inside run_latent_optimization such that the image reconstruction raises a good result.
The first try was using the naïve approach I took on previous tasks, and it was calling it inside the dedicated section as described inside the Colab notebook. The results brought using this approach were odd looking, and, while watching the process as it goes, I noticed that the mask becomes grey, and realized it happened due to the performing down sampling to the synthesized image, which also changes the results. Another approach I took, was trying to manipulate the mask image bring a black mask, but this approach, again, brought some troubling looking results. Finally, after some time playing with the code, I figured out that maybe the thing that I needed to change was the point in the code where I call the degrading function, instead of changing the degrading function itself. I figured out that the location which brings the wanted results was after the down sampling, before calculating the loss.
## Fei Fei's Inpainting
tar = "/content/gdrive/MyDrive/input_img/inputs/fei_fei_li_original.png"
outdir = "/content/gdrive/MyDrive/input_img/fei"
fei_loss = invert_image(INPAINTING_DEGRADATION,tar,outdir,"Fei Fei",num_steps=800)
## Fei Fei's Inpainting
tar = "/content/gdrive/MyDrive/input_img/inputs/fei_fei_li_original.png"
outdir = "/content/gdrive/MyDrive/input_img/fei/with 1200 and 0.3"
fei_loss = invert_image(INPAINTING_DEGRADATION,tar,outdir,"Fei Fei",num_steps=800, latent_dist_reg_weight=0.1)
## Tzlil's reconstruction:
tar = "/content/gdrive/MyDrive/input_img/Tzlil/africa_tzlil_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tzlil/Inpainting/with 800 and 0.1"
tzlil_loss = invert_image(INPAINTING_DEGRADATION, tar,outdir, "Tzlil Masking",num_steps=800,latent_dist_reg_weight=0.1)
## Tzlil's reconstruction:
tar = "/content/gdrive/MyDrive/input_img/Tzlil/africa_tzlil_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tzlil/Inpainting/with 800 and 0.001"
tzlil_loss = invert_image(INPAINTING_DEGRADATION, tar,outdir, "Tzlil Masking",num_steps=800)
## Tirtza's reconstruction:
tar = "/content/gdrive/MyDrive/input_img/Tirtsa/tir_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tirtsa/Inpainting/ with 800 and 0.1"
tirtz_loss = invert_image(INPAINTING_DEGRADATION, tar,outdir, "Tirtsa Inpainting",num_steps=800,latent_dist_reg_weight=0.1)
## Tirtza's reconstruction:
tar = "/content/gdrive/MyDrive/input_img/Tirtsa/tir_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tirtsa/Inpainting/ with 800 and 0.01"
tirtz_loss = invert_image(INPAINTING_DEGRADATION, tar,outdir, "Tirtsa Inpainting",num_steps=800,latent_dist_reg_weight=0.01)
## Tirtza's reconstruction:
tar = "/content/gdrive/MyDrive/input_img/Tirtsa/tir_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tirtsa/reconstruction/ with 1000 and 0.001"
tirtz_loss = invert_image("", tar,outdir, "reconstruction",num_steps=1000,latent_dist_reg_weight=0.001)
## Tirtza's reconstruction:
tar = "/content/gdrive/MyDrive/input_img/Tirtsa/tir_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tirtsa/Inpainting/ test"
tirtz_loss = invert_image(INPAINTING_DEGRADATION, tar,outdir, "Tirtsa Inpainting",num_steps=1600,latent_dist_reg_weight=0.7)
## Tirtza's reconstruction:
tar = "/content/gdrive/MyDrive/input_img/Tirtsa/tir_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tirtsa/reconstruction/ with 1000 and 0.1"
tirtz_loss = invert_image("", tar,outdir, "reconstruction",num_steps=1000,latent_dist_reg_weight=0.1)
## Tzlil's reconstruction:
tar = "/content/gdrive/MyDrive/input_img/Tzlil/africa_tzlil_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tzlil/reconstuction/with 1000 and 0.001"
tzlil_loss = invert_image("", tar,outdir, "Tzlil recuonstruction",num_steps=1000,latent_dist_reg_weight=0.1)
## Tzlil's reconstruction:
tar = "/content/gdrive/MyDrive/input_img/Tzlil/africa_tzlil_aligned.png"
outdir = "/content/gdrive/MyDrive/input_img/Tzlil/reconstuction/with 1000 and 0.001"
tzlil_loss = invert_image("", tar,outdir, "Tzlil recuonstruction",num_steps=1000)
!jupyter nbconvert Tzlil_IMPR_Ex5_Deep_Style_Image_Prior_2021_2022_.ipynb --to html
from google.colab import drive
drive.mount('/content/drive')
!jupyter nbconvert /content/drive/MyDrive/StyleGAN/Tzlil_IMPR_Ex5_Deep_Style_Image_Prior_2021_2022_.ipynb --to html